Object Outlining to Initiate a Visual Search

ABSTRACT

Methods and devices for initiating a search of an object are disclosed. In one embodiment, a method is disclosed that includes receiving sensor data from a sensor on a wearable computing device and, based on the sensor data, detecting a movement that defines an outline of an area in the sensor data. The method further includes identifying an object that is located in the area and initiating a search on the object. In another embodiment, a server is disclosed that includes an interface configured to receive sensor data from a sensor on a wearable computing device, at least one processor, and data storage comprising instructions executable by the at least one processor to detect, based on the sensor data, a movement that defines an outline of an area in the sensor data, identify an object that is located in the area, and initiate a search on the object.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.13/400,258, filed Feb. 20, 2012 now pending, which claims priority toU.S. Provisional Patent Application Ser. No. 61/583,995 filed Jan. 6,2012, both of which are hereby incorporated by reference herein in theirentirety.

BACKGROUND

Computing devices such as personal computers, laptop computers, tabletcomputers, cellular phones, and countless types of Internet-capabledevices are increasingly prevalent in numerous aspects of modern life.As computers become more advanced, augmented-reality devices, whichblend computer-generated information with the user's perception of thephysical world, are expected to become more prevalent.

To provide an augmented-reality experience, location and context-awarecomputing devices may be worn by a user as they go about various aspectsof their everyday life. Such computing devices, which are commonlyreferred to as “wearable” computing devices, are configured to sense andanalyze a user's environment, and to intelligently provide informationappropriate to the physical world being experienced by the user.

SUMMARY

In one aspect, an example method is disclosed that includes receivingvideo data from a camera on a wearable computing device and, based onthe video data, detecting a movement that defines an outline of an areain the video data. The method further includes identifying an objectthat is located in the area and initiating a search on the object.

In yet another aspect, a non-transitory computer-readable medium isdisclosed having stored therein instructions executable by a computingdevice to cause the computing device to perform the example methoddescribed above.

In still another aspect, a server is disclosed that includes aninterface configured to receive video data from a camera on a wearablecomputing device, at least one processor, and data storage comprisinginstructions. The instructions are executable by the at least oneprocessor to detect, based on the video data, a movement that defines anoutline of an area in the video data. The instructions are furtherexecutable by the at least one processor to identify an object that islocated in the area and initiate a search on the object.

These as well as other aspects, advantages, and alternatives, willbecome apparent to those of ordinary skill in the art by reading thefollowing detailed description, with reference where appropriate to theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flow chart illustrating an example method, in accordancewith an embodiment.

FIG. 2A is an image of a scene as it may be perceived by a user of anexample wearable computing device, in accordance with an embodiment.

FIG. 2B is an image of a scene as it may be perceived by a user of anexample wearable computing device while using a camera on the wearablecomputing device to record video data of a movement that defines anoutline of an area in the video data, in accordance with an embodiment.

FIG. 2C is an image of a scene as it may be perceived by a user of anexample wearable computing device while using a camera on the wearablecomputing device to record further video data of a movement that definesan outline of an area in the video data and displaying first additionalinformation associated with an object in the area, in accordance with anembodiment.

FIG. 2D is an image of a scene as it may be perceived by a user of anexample wearable computing device while using a camera on the wearablecomputing device to record further video data of a movement that definesan outline of an area in the video data and displaying second additionalinformation associated with an object in the area, in accordance with anembodiment.

FIG. 3A illustrates using edge detection to detect edges of an objectand aligning an outline with an edge of the object, in accordance withan embodiment.

FIG. 3B illustrates segmenting video data into layers and aligning theoutline with a layer, in accordance with an embodiment.

FIG. 4 illustrates an example system in which the disclosed methods maybe executed, in accordance with an embodiment.

FIGS. 5A-D illustrate three-dimensional renderings of example wearablecomputing devices, in accordance with an embodiment.

FIG. 6 is a simplified block diagram of an example wearable computingdevice, in accordance with an embodiment.

FIG. 7 is a simplified block diagram of an example server, in accordancewith an embodiment.

DETAILED DESCRIPTION

The following detailed description describes various features andfunctions of the disclosed systems and methods with reference to theaccompanying figures. In the figures, similar symbols typically identifysimilar components, unless context dictates otherwise. The illustrativesystem and method embodiments described herein are not meant to belimiting. It will be readily understood that certain aspects of thedisclosed systems and methods can be arranged and combined in a widevariety of different configurations, all of which are contemplatedherein.

1. OVERVIEW

A wearable computing device may be configured to record video data of,for example, an environment surrounding a user. While the wearablecomputing device is recording the video data, the user may wish tosearch for additional information for a particular object in theenvironment. Accordingly, the user may make a movement that defines anoutline of an area in the video data in which the object is located. Thewearable computing device (or, in some embodiments, a server incommunication with the wearable computing device) may, based on thevideo data, detect the movement. The wearable computing device (orserver) may then identify the object that is located in the area andinitiate a search on the object.

The above method may be particularly useful in an environment thatincludes a number of objects. The user may specify which of the objectsthe user wishes to search on, and may indicate the object by outliningthe object. In this manner, the user may more quickly receive the searchresults that the user desires, and computing power may not be expendedto perform searches in which the user is not interested.

2. EXAMPLE METHOD

FIG. 1 is a flow chart illustrating an example method 100, in accordancewith an embodiment. Method 100, shown in FIG. 1, presents an embodimentof a method that, for example, could be used with systems, devices, andservers described herein. Method 100 may include one or more operations,functions, or actions as illustrated by one or more of blocks 102-108.Although the blocks are illustrated in a sequential order, these blocksmay also be performed in parallel, and/or in a different order thanthose described herein. Also, the various blocks may be combined intofewer blocks, divided into additional blocks, and/or removed based uponthe desired implementation.

In addition, for the method 100 and other processes and methodsdisclosed herein, the flowchart shows functionality and operation of onepossible implementation of present embodiments. In this regard, eachblock may represent a module, a segment, or a portion of program code,which includes one or more instructions executable by a processor forimplementing specific logical functions or steps in the process. Theprogram code may be stored on any type of computer-readable medium, forexample, such as a storage device including a disk or hard drive. Thecomputer-readable medium may include a non-transitory computer-readablemedium, for example, such as computer-readable media that store data forshort periods of time like register memory, processor cache, and RandomAccess Memory (RAM). The computer-readable medium may also includenon-transitory media, such as secondary or persistent long term storage,like read only memory (ROM), optical or magnetic disks, and compact-discread only memory (CD-ROM), for example. The computer-readable media mayalso be any other volatile or non-volatile storage systems. Thecomputer-readable medium may be considered a computer-readable storagemedium, a tangible storage device, or other article of manufacture, forexample.

In addition, for the method 100 and other processes and methodsdisclosed herein, each block may represent circuitry that is wired toperform the specific logical functions in the process.

Blocks 102-108 may be executed by a wearable computing device, a server,or some combination thereof. For purposes of illustration, the method100 is described below for a server. It is to be understood, however,that one or more of blocks 102-108 may alternatively or additionally beexecuted by a wearable computing device. That is, in some embodiments,all of blocks 102-108 may be performed at the wearable computing device,all of blocks 102-108 may be performed at the server, or one or more ofblocks 102-108 may be performed at the wearable computing device and theremaining block(s) may be performed at the server.

As shown, the method 100 begins at block 102 where a server receivesvideo data from a camera on a wearable computing device. The video datamay be raw video data recorded at the camera, or may be video data thatis derived from raw video data recorded at the camera. Any processing ofthe raw video data may be performed at the wearable computing deviceand/or at a peripheral device or other entity that is communicativelycoupled to the wearable computing device.

The method 100 continues at block 104 where, based on the video data,the server detects a movement that defines an outline of an area in thevideo data. The movement may be, for example, a movement of a pointingobject, such as a body part of a user (e.g., a finger) or a non-bodypart controlled by a user (e.g., a stylus). Other pointing objects arepossible as well.

The wearable computing device may detect the movement based on the videodata. To this end, the wearable computing device may, for example,monitor the video data for the appearance of the pointing object. Tothis end, the pointing object may have a known and/or recognizableshape, texture, color, depth, and/or pattern. For example, inembodiments where the pointing object is a finger of the user, thewearable computing device may monitor the video data for the appearanceof an object or item having a color or tone that matches or resemblesone or more predetermined colors and/or tones considered to berepresentative of a variety of skin colors. As another example, inembodiments where the pointing object is a stylus, the wearablecomputing device may monitor the video data for the appearance of anobject or item having a color, pattern, depth, or shape that matches orresembles a known color or pattern of the stylus. For instance, ahistogram of oriented gradients (HOG) technique may be used to track thestylus (e.g., as described in “Histogram of Oriented Gradients,”Wikipedia, (Feb. 15, 2012),http://en.wikipedia.org/wiki/Histogram_of_oriented_gradients), in whichthe wearable computing device monitors the video data for the appearanceof a histogram (e.g., of intensity gradients and/or edge directions)known to describe the stylus. As still another example, the wearablecomputing device may use background subtraction techniques to detect thepointing object. To this end, the wearable computing device may compareeach frame of the video data with an “average” frame in an effort todetect changes between each frame and the average frame. The averageframe may be, for example, the first frame in the video data, the frameimmediately before each frame, or an average (e.g., a running average, arunning Gaussian average, etc.) of two or more frames before each frame.A two-dimensional convolution may be used to align the frames beforeaveraging in order to compensate for any small head movements of theuser. Other average frames are possible as well. In order to detectchanges between each frame and the average frame, the wearable computingdevice may “subtract” each frame from the average frame, such thatcommon pixels cancel and only differing pixels remain, indicatingchanges between each frame and the average frame. The changes may, forexample, indicate movement of the pointing object. Other techniques maybe used as well, including, for example, those described in thefollowing papers and publications: Oikonomidis et al., “EfficientModel-based 3D Tracking of Hand Articulations using Kinect,” The 22^(nd)British Machine Vision Conference (August 2011); Wang et al., “Real-TimeHand-Tracking with a Color Glove,” ACM Transactions on Graphics (July2009); Lyons et al., “GART: The Gesture and Activity RecognitionToolkit,” HCl (July 2007); de la Hamette et al., “Laser Triangulation asa means of robust Visual Input for Wearable Computers,” InternationalSymposium on Wearable Computers (October 2004); Athitsos et al.,“Estimating 3D Hand Pose from a Cluttered Image,” Proceedings of theIEEE Computer Society Conference on Computer Vision and PatternRecognition (June 2003); Stenger et al., “Model-Based 3D Tracking of anArticulated Hand,” Proceedings of the IEEE Computer Society Conferenceon Computer Vision and Pattern Recognition (December 2001); and Freemanet al., “Orientation Histograms for Hand Gesture Recognition,”Mitsubishi Electric Research Laboratories (December 1994). It will beappreciated that the movement may be detected from the video data usingother techniques as well.

In some embodiments, the wearable computing device may be configuredwith lenses on which the video data is displayed, such that the usersees representations of the area, the object, and the pointing devicedisplayed by the wearable computing device. In these embodiments, theoutline that the user sees the user to be defining may be substantiallysimilar to the outline that the server determines the user to bedefining based on the video data. Thus, in these embodiments, the servermay detect the movement by simply tracking the pointing device.

In other embodiments, however, the wearable computing device may beconfigured with substantially transparent lenses, such that the usersees the area, the object, and the pointing object in the real world. Inthese embodiments, the camera on the wearable computing device may notbe perfectly aligned with an eye of the user, such that the outline thatthe user sees the user to be defining may be shifted from the outlinethat the wearable computing device determines the user to be defining.In order to correct for this shift, the server may determine a distancebetween the user's eye and the camera and/or an angle between the user'sline of sight and the camera's alignment (e.g., by receiving thedistance and/or the angle from the wearable computing device) and mayuse the distance and/or the angle to account for the shift. Thus, inthese embodiments, the server may detect the movement by tracking thepointing device and use the distance and/or the angle to account for theshift.

At block 106, the server identifies an object that is located in thearea. To this end, the server may assume the object substantially fillsthe area, such that the outline of the area is also the outline of theobject. Alternatively or additionally, the server may, in someembodiments, use edge detection to detect edges of the object and alignthe outline with at least one edge of the object. This use of edgedetection is further described below in connection with FIG. 3A.Alternatively or additionally, the server may, in some embodiments,segment the video data into layers and align the outline with at leastone layer. This use of segmenting is further described below inconnection with FIG. 3B. The server may identify the object in othermanners as well.

At block 108, the server initiates a search on the object. The searchmay be, for example, a visual search, such as a search of a two- orthree-dimensional image or model database. The database may be stored atthe wearable computing device and/or the server, or may be storedremotely and accessible to the wearable computing device and/or theserver. Searching the database may involve, for example, comparing theobject (or the area that includes the object) with some or all of theimages or models in the database. The server may then select an image ormodel that is most similar to the object (or area). Similarity may bedetermined based on, for example, a number or configuration of visualfeatures (e.g., colors, shapes, textures, brightness levels, etc.) inthe object (or area) and the object-model. The search may take otherforms as well.

In some embodiments, the server may initiate the search before themovement is complete and may periodically update the search during themovement. As more of the outline is defined, the server may betteridentify the object and, in turn, may more effectively search thedatabase. The server may update the search in response to one or both ofreceiving additional video data from the wearable computing device anddetecting additional movement.

Once the search is initiated, the server may cause the wearablecomputing device to display information corresponding to results of thesearch. The information may be, for example, textual and/or graphicalinformation related to the object, such as a top result of the search.Other information is possible as well.

In some embodiments, the wearable computing device may determine,generate, or search for the additional information at the time ofselecting each image or model, and the wearable computing device maydisplay the additional information as the image or model is selected.Thus, as the user outlines the area, as described above, the user maysee the additional information associated with the images or modelsselected during the search.

In still other embodiments, the wearable computing device may determine,generate, or search for the additional information at the time ofselecting each image or model, but rather than displaying the additionalinformation, the wearable computing device may cache the information.The additional information may then be read from the cache and displayedin response to, for example, detecting that the user has completed theoutlining, or in response to another trigger.

3. EXAMPLE EMBODIMENT

FIG. 2A is an image of a scene 200 as it may be perceived by a user ofan example wearable computing device, in accordance with an embodiment.The scene 200 may be a frame from video data recorded by a camera on thewearable computing device and displayed on lenses of the wearablecomputing device. The video data may be raw video data recorded at thecamera, or may be video data that is derived from raw video datarecorded at the camera. Alternatively, the scene 200 may be a real-worldscene visible to the user through transparent lenses on the wearablecomputing device. As shown, the scene 200 includes a first object 202and a second object 204.

FIG. 2B is an image of the scene 200 as it may be perceived by the userof the example wearable computing device while using the camera on thewearable computing device to record video data of a movement thatdefines an outline 208 of an area in the video data, in accordance withan embodiment. As described above, a wearable computing device and/or aserver may, based on the video data, detect the movement defining theoutline 208 of an area by, for example, tracking the movement of apointing device 206. While the pointing device 206 is shown as a finger,other pointing devices are possible as well, as described above.

The outline 208 may be displayed by the wearable computing device so asto provide feedback to the user while the user performs the movement.While the outline 208 is shown as a solid line, in other embodiments theoutline 208 may be indicated in other ways, such as, for example, as adotted line, a colored line, a line that appears to “cut” the firstobject 202 out of the scene 200, or another type of line. In someembodiments, the outline 208 may not be displayed at all.

In response to detecting the movement defining the outline 208 of thefirst object 202, the wearable computing device and/or server mayinitiate a search on the first object 202, as described above. Further,the wearable computing device may display information corresponding toresults of the search.

FIG. 2C is an image of the scene 200 as it may be perceived by the userof the example wearable computing device while using the camera on thewearable computing device to record further video data of a movementthat defines the outline 208 of the area in the video data anddisplaying first additional information 210 associated with the firstobject 202 in the area, in accordance with an embodiment. As shown inFIG. 2C, the first additional information 210 is overlaid on the scene200. In other embodiments, the first additional information 210 may bedisplayed in other manners as well. For example, the first additionalinformation 210 may scroll across the scene 200, may replace the scene200, or may be displayed on a separate display from the scene 200. Instill other embodiments, the first additional information 210 may bepresented to the user as audio information. The first additionalinformation 210 may take other forms as well. Further, while the firstadditional information 210 is shown to include certain information, thefirst additional information may include any information related to thefirst object 202.

As described above, the search may be periodically updated as theoutline 208 is further defined by the pointing object 206. FIG. 2D is animage of the scene 200 as it may be perceived by the user of the examplewearable computing device while using the camera on the wearablecomputing device to record further video data of the movement thatdefines the outline 208 of the area in the video data and displayingsecond additional information 212 associated with the first object 202in the area, in accordance with an embodiment. In some cases, as aresult of the outline 208 being further defined, the second additionalinformation 212 may more accurately represent the first object 202. Thesecond additional information 212 may be displayed in any of the mannersdescribed above for the first additional information 210, and may takeany of the forms described above for the first additional information210.

While the foregoing description focused on embodiments in which thesearch is initiated before the movement is complete and updated duringthe movement, in other embodiments, the search may not be initiateduntil the movement is complete (e.g., in response to detecting that thepointing object 206 has stopped moving and/or detecting that thepointing object 206 has returned to its initial position, therebyclosing the outline 208, etc.) and/or may not be updated during themovement.

4. EXAMPLE TECHNIQUES FOR DETECTING AN OBJECT IN AN AREA

As noted above, in order to identify an object that is located in anarea whose outline is defined by a movement, as described above, awearable computing device and/or server may assume the objectsubstantially fills the area, such that the outline of the area is alsothe outline of the object. Alternatively or additionally, the server mayuse one or more techniques to identify the object including, forexample, edge-detection and/or segmenting techniques.

FIG. 3A illustrates using edge detection to detect edges 306 of anobject 302 and aligning an outline 304 with an edge 306 of the object302, in accordance with an embodiment. As shown, the outline 304 isdefined for the object 302 in, for example, any of the manners describedabove. Further, edges 306 of the object are detected. The wearablecomputing device and/or server may detect the edges 306 by, for example,scanning the scene 300 for discontinuities, such as discontinuities indepth, surface orientation, color, and/or brightness, and may detect theedges to be along the discontinuities. The edge detection may be furtherrefined using one or more techniques including, for example,search-based edge detection, zero-crossing-based edge detection, andCanny edge detection. Other edge-detection techniques are possible aswell.

Once the server and/or the wearable computing device detects the edges306, the server and/or the wearable computing device may align theoutline 304 to at least one edge 306 of the object 302. This aligningmay aid the server and/or wearable computing device in identifying theobject 302.

FIG. 3B illustrates segmenting video data into layers and aligning theoutline 304 with a layer 308, in accordance with an embodiment. Asshown, the outline 304 is defined for the object (not shown) in, forexample, any of the manners described above. Further, the scene 300 issegmented into a number of layers. The wearable computing device and/orserver may segment the scene 300 by, for example, segmenting the scene300 into layers that have common depth, surface orientation, color,and/or brightness. Other segmenting techniques are possible as well. Asshown, a layer 308 substantially includes the object.

Once the server and/or the wearable computing device segments the scene300 into layers, the server and/or the wearable computing device mayalign the outline 304 with the layer 308 that substantially includes theobject 302. This aligning may aid the server and/or wearable computingdevice in identifying the object 302.

The server and/or the wearable computing device may identify the objectusing other techniques as well.

5. EXAMPLE ARCHITECTURE

Systems and devices in which example embodiments of the above examplemethods may be implemented will now be described in greater detail. Ingeneral, an example system may be implemented in or may take the form ofa wearable computing device. However, an example system may also beimplemented in or take the form of other devices, such as a mobile phoneor tablet computer, among others. Further, an example system may takethe form of non-transitory computer-readable medium, which has programinstructions stored thereon that are executable by at least oneprocessor to provide the functionality described herein. An examplesystem may also take the form of a device, such as a wearable computingdevice, mobile phone, or tablet computer, or a subsystem of such adevice that includes such a non-transitory computer-readable mediumhaving such program instructions stored thereon.

a. Example System

FIG. 4 illustrates an example system 400 in which the disclosed methodsmay be executed, in accordance with an embodiment. As shown, the system400 includes a wearable computing device 402 that is communicablycoupled to a server 404 via a wireless link 406. While only one wearablecomputing device 402 and one server 404 is shown, more or fewer wearablecomputing devices and/or servers are possible as well.

As shown, the wearable computing device 402 is a pair of glasses. Inother embodiments, however, other computing devices could additionallyor alternatively be used. For example, the wearable computing device 402may be an otherwise head-mounted device, such as a visor, headphones, ahat, a headband, an earpiece, or any other type of headwear that isconfigured to communicably couple to the server 402. Alternatively oradditionally, the wearable computing device 402 could be anotherwise-wearable computing device, such as a backpack, fanny pack,belt, or any other piece of body wear that is configured to communicablycouple to the server. Still alternatively or additionally, the wearablecomputing device 402 may be a non-wearable computing device, such as amobile phone, a tablet computer, or any other device configured tocommunicably couple to server 404.

The server 404 may be, for example, a computer or plurality of computerson which one or more programs and/or applications are executed in orderto provide one or more wireless and/or web-based interfaces that areaccessible by the wearable computing device 402 via the wireless link406.

The wireless link 406 may use, for example, Bluetooth® radio technology,communication protocols described in IEEE 802.11 (including any IEEE802.11 revisions), cellular technology (such as GSM, CDMA, UMTS, EV-DO,WiMAX, or LTE), or Zigbee® technology, among other possibilities.Alternatively or additionally, the wireless link 406 may be a wiredlink, such as a wired serial bus (e.g., a universal serial bus or aparallel bus). In either case, the wireless link 406 may be aproprietary connection.

An example wearable computing device is further described below inconnection with FIGS. 5A-D and 6, while an example server is furtherdescribed below in connection with FIG. 7.

b. Example Wearable Computing Device

FIGS. 5A-D illustrate three-dimensional renderings of example wearablecomputing devices, in accordance with an embodiment. As shown in FIG.5A, the wearable computing system takes the form of a head-mounteddevice 500 (which may also be referred to as a head-mounted display).The wearable computing device may take other forms as well, includingany of those described above.

As illustrated in FIG. 5A, the head-mounted device 500 comprises frameelements including lens-frames 502, 504 and a center frame support 506,lens elements 508, 510, and extending side-arms 512, 514. The centerframe support 506 and the extending side-arms 512, 514 are configured tosecure the head-mounted device 500 to a user's face via a user's noseand ears, respectively.

Each of the frame elements 502, 504, and 506 and the extending side-arms512, 514 may be formed of a solid structure of plastic and/or metal, ormay be formed of a hollow structure of similar material so as to allowwiring and component interconnects to be internally routed through thehead-mounted device 500. Other materials may be possible as well.

One or more of each of the lens elements 508, 510 may be formed of anymaterial that can suitably display a projected image or graphic. Each ofthe lens elements 508, 510 may also be sufficiently transparent to allowa user to see through the lens element. Combining these two features ofthe lens elements 508, 510 may facilitate an augmented reality orhead-up display where a projected image or graphic is superimposed overa real-world view as perceived by the user through the lens elements.For example, combining these two features of the lens elements 508, 510may allow the head-mounted device 510 to overlay an object-model on anobject, as described above.

The extending side-arms 512, 514 may each be projections that extendaway from the lens-frames 508, 510, respectively, and may be positionedbehind a user's ears to secure the head-mounted device 500 to the user.The extending side-arms 512, 514 may further secure the head-mounteddevice 500 to the user by extending around a rear portion of the user'shead (not shown). Additionally or alternatively, for example, thehead-mounted device 500 may connect to or be affixed within ahead-mounted helmet structure (not shown). Other possibilities exist aswell.

The head-mounted device 500 may also include an on-board computingsystem 516, a video camera 518, a sensor 520, and a finger-operabletouch pad 522. The on-board computing system 516 is shown to bepositioned on the extending side-arm 512 of the head-mounted device 500;however, the on-board computing system 516 may be provided on otherparts of the head-mounted device 100 or may be positioned remote fromthe head-mounted device 500 (e.g., the on-board computing system 516could be wire- or wirelessly-connected to the head-mounted device 500).The on-board computing system 516 may include a processor and memory,for example. The on-board computing system 516 may be configured toreceive and analyze data from the video camera 518 and thefinger-operable touch pad 522 (and possibly from other sensory devices,user interfaces, or both) and generate images for output by the lenselements 508 and 510.

The video camera 518 is shown positioned on the extending side-arm 512of the head-mounted device 500; however, the video camera 518 may beprovided on other parts of the head-mounted device 500. The video camera518 may be configured to capture images at various resolutions or atdifferent frame rates. A number of types of video cameras with a smallform-factor, such as those used in cell phones or webcams, for example,may be incorporated into an example of the head-mounted device 500.

Further, although FIG. 5A illustrates one video camera 518, more videocameras may be used, and each may be configured to capture the sameview, or to capture different views. For example, the video camera 518may be forward facing to capture at least a portion of the real-worldview perceived by the user. This forward facing image captured by thevideo camera 518 may then be used to generate an augmented reality inwhich computer generated images appear to interact with the real-worldview perceived by the user.

The sensor 520 is shown on the extending side-arm 514 of thehead-mounted device 500; however, the sensor 520 may be positioned onother parts of the head-mounted device 500. The sensor 520 may includeone or more of a gyroscope or an accelerometer, for example. Othersensing devices may be included within, or in addition to, the sensor520 or other sensing functions may be performed by the sensor 520.

The finger-operable touch pad 522 is shown on the extending side-arm 512of the head-mounted device 500. However, the finger-operable touch pad522 may be positioned on other parts of the head-mounted device 500.Also, more than one finger-operable touch pad may be present on thehead-mounted device 500. The finger-operable touch pad 522 may be usedby a user to input commands. The finger-operable touch pad 522 may senseat least one of a position and a movement of a finger via capacitivesensing, resistance sensing, or a surface acoustic wave process, amongother possibilities. The finger-operable touch pad 522 may be capable ofsensing finger movement in a direction parallel or planar to the padsurface, in a direction normal to the pad surface, or both, and may alsobe capable of sensing a level of pressure applied to the pad surface.The finger-operable touch pad 522 may be formed of one or moretranslucent or transparent insulating layers and one or more translucentor transparent conducting layers. Edges of the finger-operable touch pad522 may be formed to have a raised, indented, or roughened surface, soas to provide tactile feedback to a user when the user's finger reachesthe edge, or other area, of the finger-operable touch pad 522. If morethan one finger-operable touch pad is present, each finger-operabletouch pad may be operated independently, and may provide a differentfunction.

FIG. 5B illustrates an alternate view of the three-dimensional renderingof the example wearable computing device illustrated in FIG. 5A, inaccordance with an embodiment. As shown in FIG. 5B, the lens elements508, 510 may act as display elements. The head-mounted device 500 mayinclude a first projector 524 coupled to an inside surface of theextending side-arm 514 and configured to project a display 528 onto aninside surface of the lens element 510. Additionally or alternatively, asecond projector 526 may be coupled to an inside surface of theextending side-arm 512 and configured to project a display 530 onto aninside surface of the lens element 508.

The lens elements 508, 510 may act as a combiner in a light projectionsystem and may include a coating that reflects the light projected ontothem from the projectors 524, 526. In some embodiments, a reflectivecoating may not be used (e.g., when the projectors 524, 526 are scanninglaser devices).

In alternative embodiments, other types of display elements may also beused. For example, the lens elements 508, 510 themselves may include: atransparent or semi-transparent matrix display, such as anelectroluminescent display or a liquid crystal display, one or morewaveguides for delivering an image to the user's eyes, or other opticalelements capable of delivering an in focus near-to-eye image to theuser. A corresponding display driver may be disposed within the frameelements 508, 510 for driving such a matrix display. Alternatively oradditionally, a laser or LED source and scanning system could be used todraw a raster display directly onto the retina of one or more of theuser's eyes. Other possibilities exist as well.

FIG. 5C illustrates a three-dimensional rendering of another wearablecomputing system, in accordance with an embodiment. As shown in FIG. 5C,the wearable computing device takes the form of a head-mounted device500. The head-mounted device 500 may include frame elements andside-arms similar to those described above in connection with FIGS.5A-B. The head-mounted device 500 may additionally include an on-boardcomputing system 516 and a video camera 518, which may take any of theforms described above in connection with FIGS. 5A-B. The video camera518 is shown mounted on a frame of the head-mounted device 500. However,the video camera 518 may be mounted at other positions as well.

As shown in FIG. 5C, the head-mounted device 500 may include a singledisplay 532 which may be coupled to the head-mounted device 500. Thedisplay 532 may be formed on one of the lens elements of thehead-mounted device 500 and may be configured to overlaycomputer-generated graphics in the user's view of the physical world.For example, the display 532 may be configured to overlay anobject-model on an object in the physical world, as described above. Thedisplay 532 is shown to be provided in a center of a lens of thehead-mounted device 500; however, the display 532 may be provided inother positions as well. The display 532 is controllable via thecomputing system 516, which may be coupled to the display 532 via anoptical waveguide 534, as shown.

FIG. 5D illustrates a three-dimensional rendering of another wearablecomputing system, in accordance with an embodiment. As shown in FIG. 5D,the wearable computing device takes the form of a head-mounted device500. The head-mounted device 500 may include side-arms 512, 514 and acenter frame support 506, which take any of the forms described above inconnection with FIGS. 5A-B. Further, the head-mounted device 500 mayinclude a bridge portion with nosepiece 542. The head-mounted device 500may additionally include an on-board computing system and a video camera(not shown), which may take any of the forms described above inconnection with FIGS. 5A-B.

As shown in FIG. 5D, the center frame support 506 connects the side-arms512, 514. In contrast to those shown in FIGS. 5A-C, the head-mounteddevice 500 shown in FIG. 5D does not include lens-frames containing lenselements. Rather, as shown, the head-mounted device 500 may include asingle lens element 540 that may be coupled to one of the side-arms 512and/or the center frame support 506. The single lens element 540 mayinclude a display and may be configured to overlay computer-generatedgraphics upon the user's view of the physical world. For example, thedisplay 532 may be configured to overlay an object-model on an object inthe physical world, as described above. The display may be controllablevia the computing system, which may be coupled to the display via anoptical waveguide 538, as shown.

In one example, the single lens element 540 may be coupled to the innerside (i.e., the side exposed to a portion of a user's head when worn bythe user) of the extending side-arm 512. The single lens element 540 maybe positioned in front of or proximate to a user's eye when thehead-mounted device 500 is worn by a user. For example, the single lenselement 540 may be positioned below the center frame support 506, asshown in FIG. 5D.

FIG. 6 is a simplified block diagram of an example wearable computingdevice 600, in accordance with an embodiment. As shown, the wearablecomputing device 600 includes a camera 602, a display 604, a serverinterface 606, a processor 608, and data storage 610, all of which maybe communicatively linked together by a system bus, network, and/orother connection mechanism 612.

The camera 602 may be any camera configured to record video data. Tothis end, the camera 602 may be configured to detect visible light, ormay be configured to detect light from other portions of the spectrum,such as infrared or ultraviolet light, or x-rays. Other types of camerasare possible as well. The camera 602 may be a two-dimensional detector,or may have a three-dimensional spatial range. In some embodiments, thecamera 602 may be, for example, a range detector configured to generatea two-dimensional image showing a distance from the camera 602 to anumber of points in the video data. To this end, the camera 602 may useone or more range detecting techniques. For example, the camera 602 mayuse a structured light technique in which the wearable computing device600 illuminates an object with a predetermined light pattern, such as agrid or checkerboard pattern and uses the camera 602 to detect areflection of the predetermined light pattern off the object. Based ondistortions in the reflected light pattern, the wearable computingdevice 600 may determine the distance to the points on the object. Thepredetermined light pattern may comprise infrared light, or light ofanother wavelength. As another example, the camera 602 may use a laserscanning technique in which the wearable computing device 600 emits alaser and scans across a number of points the object. While scanning theobject, the wearable computing device 600 uses the camera 602 to detecta reflection of the laser off the object for each point. Based on alength of time it takes the laser to reflect off the object at eachpoint, the wearable computing device 600 may determine the distance tothe points on the object. As yet another example, the camera 602 may usea time-of-flight technique in which the wearable computing device 600emits a light pulse and uses the camera 602 to detect a reflection ofthe light pulse off an object at a number of points on the object. Inparticular, the camera 602 may include a number of pixels, and eachpixel may detect the reflection of the light pulse from a point on theobject. Based on a length of time it takes the light pulse to reflectoff the object at each point, the wearable computing device 600 maydetermine the distance to the points on the object. The light pulse maybe a laser pulse. Other range detecting techniques are possible as well,including stereo triangulation, sheet-of-light triangulation,interferometry, and coded aperture techniques, among others. In someembodiments, the camera 602 may be enhanced through sensor fusiontechnology. The camera 602 may further take any of the forms describedabove in connection with FIG. 5A.

The display 604 may be any display configured to display an outlinedefined by a movement and/or additional information related to anobject, as described above. In some embodiments, the display mayadditionally be configured to display the video data. To this end, thedisplay 604 may be configured to display information received from theprocessor 608. The display 604 may additionally be configured to displayinformation received from one or more additional sources. The display604 may be, for example, a head-up display, a head-mounted display, anoptical see-through display, an optical see-around display, a videosee-through display, a flat-panel display, a light-emitting diode (LED)display, an electroluminescent display (ELD), a liquid crystal display(LCD), an organic LED (OLED) display, or any other type of display nowknown or later developed. The display 604 may alternatively oradditionally take any of the forms described above in connection withFIGS. 5A-D.

The server interface 606 may be any interface configured to wirelesslycommunicate with a server. In particular, the server interface 606 maybe configured to transmit to the server one or more of the video data(or data derived from the video data) recorded by the camera 602, arequest to initiate a search, and an updated request to update thesearch. Further, the server interface 606 may be configured to receivefrom the server information corresponding to results of the search.

To this end, the server interface 606 may include an antenna and achipset for communicating with the server over an air interface. Thechipset or server interface 606 in general may be arranged tocommunicate according to one or more other types of wirelesscommunication (e.g. protocols) such as Bluetooth, communicationprotocols described in IEEE 802.11 (including any IEEE 802.11revisions), cellular technology (such as GSM, CDMA, UMTS, EV-DO, WiMAX,or LTE), or Zigbee, among other possibilities. In some embodiments, theserver interface 606 may also be configured to wirelessly communicatewith one or more other devices, such as a database and/or other wearablecomputing devices.

The processor 608 may comprise one or more general-purpose processorsand/or one or more special-purpose processors. To the extent theprocessor 608 includes more than one processor, such processors couldwork separately or in combination. Further, the processor 608 may beintegrated in whole or in part with the server interface 604 and/or withother components.

Data storage 610, in turn, may comprise one or more volatile and/or oneor more non-volatile storage components, such as optical, magnetic,and/or organic storage, and data storage 610 may be integrated in wholeor in part with the processor 608. In some embodiments, data storage 610may contain instructions 614 (e.g., program logic) executable by theprocessor 608 to execute various wearable computing device functions.For example, data storage 610 may contain instructions 614 executable bythe processor 608 to cause the camera 602 to record video data and,based on the video data, detect a movement that defines an outline of anarea. Further, data storage 610 may contain instructions 614 executableby the processor 608 to identify an object that is located in the areaand initiate a search on the object. This identifying may involve thewearable computing device 600 searching a database itself, or mayinvolve the wearable computing device 600 causing the server interface606 to transmit to the server a request for the server to search thedatabase. The request may include the video data (or data derived fromthe video data) recorded by the camera 602. Data storage 610 may containadditional instructions 614 as well.

The wearable computing device 600 may include one or more elements inaddition to or instead of those shown. For example, the wearablecomputing device 600 may include one or more additional interfacesand/or one or more power supplies. Other additional components arepossible as well. In these embodiments, data storage 610 may furtherinclude instructions executable by the processor(s) to control and/orcommunicate with the additional components.

c. Example Server

FIG. 7 shows a block diagram of an example server 700, in accordancewith an embodiment. As shown, the server 700 includes a device interface702, a processor 704, and data storage 706, all of which may becommunicatively linked together by a system bus, network, and/or otherconnection mechanism 708.

The device interface 702 may be any interface configured to wirelesslycommunicate with a wearable computing device. In particular, the deviceinterface 702 may be configured to receive from the wearable computingdevice one or more of video data (or data derived from video data)recorded by a camera at the wearable computing device, a request toinitiate search, and an updated request to update the search. Further,the device interface 702 may be configured to transmit to the wearablecomputing device additional information corresponding to results of thesearch.

To this end, the device interface 702 may include an antenna and achipset for communicating with the wearable computing device over an airinterface. The chipset or device interface 702 in general may bearranged to communicate according to one or more other types of wirelesscommunication (e.g. protocols) such as Bluetooth, communicationprotocols described in IEEE 802.11 (including any IEEE 802.11revisions), cellular technology (such as GSM, CDMA, UMTS, EV-DO, WiMAX,or LTE), or Zigbee, among other possibilities. In some embodiments, thedevice interface 702 may also be configured to wirelessly communicatewith one or more other devices, such as a database and/or other wearablecomputing devices.

The processor 704 may comprise one or more general-purpose processorsand/or one or more special-purpose processors. To the extent theprocessor 704 includes more than one processor, such processors couldwork separately or in combination. Further, the processor 704 may beintegrated in whole or in part with the device interface 702 and/or withother components.

Data storage 706, in turn, may comprise one or more volatile and/or oneor more non-volatile storage components, such as optical, magnetic,and/or organic storage, and data storage 706 may be integrated in wholeor in part with the processor 706. As shown, data storage 706 maycontain instructions 712 (e.g., program logic) executable by theprocessor 706 to execute various server functions. For example, datastorage 706 may include instructions 712 executable by the processor 706to detect, based on the video data received from the wearable computingdevice via the device interface 702, a movement that defines an outlineof an area in the video data. In some embodiments, the server 700 mayitself detect the movement, while in other embodiments the wearablecomputing device may detect the movement and the video data receivedfrom the wearable computing device may include an indication of thedetected movement. The data storage 706 may further include instructions712 executable by the processor 706 to identify an object that islocated in the area and initiate a search on the object. Data storage706 may include additional program logic as well.

The server 700 may include one or more elements in addition to orinstead of those shown. For example, the server 700 may include one ormore additional interfaces and/or one or more power supplies. As anotherexample, the server 700 may include database in which the search takesplace. The database may take any of the forms described above. Otheradditional components are possible as well. In these embodiments, datastorage 706 may further include instructions executable by theprocessor(s) to control and/or communicate with the additionalcomponents.

6. CONCLUSION

While various aspects and embodiments have been disclosed herein, otheraspects and embodiments will be apparent to those skilled in the art.The various aspects and embodiments disclosed herein are for purposes ofillustration and are not intended to be limiting, with the true scopeand spirit being indicated by the following claims.

1. A method comprising: receiving sensor data from a sensor on awearable computing device; based on the sensor data, detecting amovement that defines an outline of an area in the sensor data;identifying an object that is located in the area; and initiating asearch on the object.
 2. The method of claim 1, further comprisingcausing the wearable computing device to display informationcorresponding to results of the search.
 3. The method of claim 1,wherein initiating the search comprises: beginning the search before themovement is complete; and periodically updating the search during themovement.
 4. The method of claim 3, wherein: beginning the searchcomprises causing the wearable computing device to provide informationcorresponding to results of the search; and periodically updating thesearch comprises causing the wearable computing device to provideupdated information corresponding to results of the updated search. 5.The method of claim 1, further comprising: during the search, cachinginformation corresponding to results of the search; and causing thewearable computing device to provide at least some of the cachedinformation.
 6. The method of claim 1, wherein the search comprises avisual search.
 7. The method of claim 1, wherein: the movement comprisesmovement of a pointing device; and detecting the movement comprisestracking the pointing device.
 8. The method of claim 7, wherein thepointing device comprises a finger.
 9. The method of claim 1 wherein:the sensor is configured to be offset from an eye of a user by an angleand a distance; and detecting the movement is further based on the angleand the distance.
 10. The method of claim 1, wherein identifying theobject comprises: using edge detection to detect edges of the object;and aligning the outline with at least one edge of the object.
 11. Themethod of claim 1, wherein identifying the object comprises: segmentingthe sensor data into layers; and aligning the outline with at least onelayer.
 12. A wearable computing device comprising: an interfaceconfigured to receive sensor data from a sensor on a wearable computingdevice; at least one processor; and data storage comprising instructionsexecutable by the at least one processor to: based on the sensor data,detect a movement that defines an outline of an area in the sensor data;identify an object that is located in the area; and initiate a search onthe object.
 13. The wearable computing device of claim 12, furthercomprising a speaker configured to provide audio informationcorresponding to results of the search.
 14. The wearable computingdevice of claim 13, wherein the device comprises a head-mountabledevice.
 15. The wearable computing device of claim 12, whereinperforming the search comprises: beginning the search before themovement is complete; and periodically updating the search during themovement.
 16. The wearable computing device of claim 12, wherein: themovement comprises movement of a pointing device; and detecting themovement comprises tracking the pointing device.
 17. A non-transitorycomputer-readable medium having stored therein instructions executableby a computing device to cause the computing device to perform functionscomprising: receiving sensor data from a sensor of a wearable computingdevice; based on the sensor data, detecting a movement that defines anoutline of an area in the sensor data; identifying an object that islocated in the area; and initiating a search on the object.
 18. Thenon-transitory computer-readable medium of claim 17, the functionsfurther comprising causing the wearable computing device to provideinformation corresponding to results of the search.
 19. Thenon-transitory computer-readable medium of claim 17, wherein performingthe search comprises: beginning the search before the movement iscomplete; and periodically updating the search during the movement.